Tokenizer rules file

The file is divided in three sections <Macros>, <RegExps> and <Abbreviations>. Each section is closed by </Macros>, </RegExps> and </Abbreviations> tags respectively.

The <Macros> section allows the user to define regexp macros that will be used later in the rules. Macros are defined with a name and a Perl regexp.
E.g. ALPHA [A-Za-z]

The <RegExps> section defines the tokenization rules. Previously defined macros may be referred to with their name in curly brackets.
E.g. *ABREVIATIONS1 0 ((\{ALPHA\}+\.)+)(?!\.\.)

Rules are regular expressions, and are applied in the order of definition. The first rule matching the beginning of the line is applied, a token is built, and the rest of the rules are ignored. The process is repeated until the line has been completely processed.

• The first field in the rule is the rule name. If it starts with a *, the RegExp will only produce a token if the match is found in abbreviation list (<Abbreviations> section).
• The second field in the rule is the substring to form the token/s with It may be 0 (the match of the whole expression) or any number from 1 to the number of substrings (up to 9). A token will be created for each substring from 1 to the specified value.
• The third field is the regexp to match against the input. line. Any Perl regexp convention may be used.

The <Abbreviations> section defines common abbreviations (one per line) that must not be separated of their following dot (e.g. etc., mrs.). They must be lowercased.